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Sir: 

We, Anthony Mai and Glen Van Datta, hereby declare as follows: 

1 . We are the joint inventors of the above-noted United States Patent Application 
10/700,798, filed in the United States Patent and Trademark Office on November 3, 2003, and 
with a claim of priority under 35 U.S.C. 1 19(e) to Provisional Application 60/513,098, filed 
October 20, 2003. 
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2. We hereby declare we conceived and reduced to practice the invention defined by 
claim 24 ("the invention") of the above-noted application prior to April 9, 2002, the United 
States filing date of United States Patent 7,174,382 issued to Ramanathan et al. ("Ramanathan"), 
as demonstrated in the exhibits attached to this Declaration. Our earlier conception and 
reduction to practice of my claimed invention is evidenced by the following statements: 

3. Prior to April 9, 2002, we conceived of the invention of the present application as 
evidenced by Exhibit A, titled "Multi-Channel Multi-Party Audio Streaming Protocol" 
("Protocol"), which was attached to an e-mail that Anthony Mai sent to Glen Van Datta prior to 
April 9, 2002. Language in the e-mail portion of Exhibit A has been redacted to preserve 
attorney-client privileged information. Specific nomenclature in the Protocol has been redacted 
to preserve confidential information. 

4. The Protocol discloses the elements recited in claim 24. In particular, the Protocol 
describes the method of joining (adding) a peer system to a peer-to-peer (P2P) system and a 
method of establishing a P2P network. 

5. Our invention was reduced to practice in a computer implementation as evidenced 
by the attached Exhibits B and C, which perform the functions recited by the elements recited in 
claim 14. These exhibits are source code that is proprietary to the assignee of the present 
invention; and such source code has been redacted to preserve the confidentiality of such source 
code. 

6. Exhibit B is computer source code created prior to April 9, 2002. Exhibit B 
constructs and sends out communications packages, as well as receives and processes incoming 
communication packages, pertaining to the forming and maintenance of the relay grid. Exhibit B 
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describes the data packages that the relay grid tries to relay. Portions of Exhibit B have been 
redacted to preserve confidential information. 

7. Exhibit C is computer source code created prior to April 9, 2002. Exhibit C 
manages the features in Exhibit B as well as manages the high level application requests. Exhibit 
C generates and processes the message packages that are used to implement the invention. 
Exhibit C also accepts incoming and outgoing audio data streams and processes the data streams 
in proper data packages Exhibit C utilizes and manages Exhibit B to allow each client to interact 
with each other using pre-defined message packages in order to connect to each other and form 
the relay grid described in the invention. Function calls from Exhibit C are reproduced in 
Exhibits C1-C8 and are explained in more detail herein as necessary. Portions of Exhibits C and 
C1-C8 have been redacted to preserve confidential information. 

8. Exhibit D includes computer "screen captures" resulting from execution of the 
source code and algorithms of Exhibit B and Exhibit C in a computer environment. The 
resulting "screen captures" of Exhibit D were successful and repeatable. Exhibit D is evidence 
that our invention was reduced to practice. Portions of Exhibit D have been redacted to preserve 
confidential information. 

9. The "screen captures" of Exhibit D were prepared recently using the source code and 
algorithms of Exhibit B and Exhibit C running on a computer test platform available during the 
testing of the present invention. Such testing was performed prior to April 9, 2002 to determine 
the source code was successful, operable, and repeatable. 

10. The function call on page 10 of Exhibit C and reproduced as Exhibit CI causes the 
code to start the process to construct a relay grid, which implements the element "adding a peer 
system to a peer-to-peer relay network," recited in claim 24. 
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1 1 . The function call on page 1 1 of Exhibit C and reproduced as Exhibit C2 causes the 
code to process any incoming network package and decide further processing depending on the 
package, which implements "opening a connection between a server and a joining peer system," 
recited in claim 24. 

12. The function call on page 24 of Exhibit C and reproduced as Exhibit C3; the 
function call on page 25 of Exhibit C and reproduced as Exhibit C4; the function call on page 26 
of Exhibit C and reproduced as Exhibit C5; and the function call on page 27 of Exhibit C and 
reproduced as Exhibit C6; cause the code to allow top application layer code to obtain 
information about existing channels (relay grid) and clients who have joined in each channel, 
which implements "providing grid information to said joining peer system indicating one or 
more established peer-to-peer relay networks," recited in claim 24. 

13. The function call on page 22 of Exhibit C and reproduced as Exhibit C7 causes the 
code to cause the local client to join a relay grid, which implements "receiving a grid selection 
from said joining peer system indicating a selected peer-to-peer relay network, wherein said 
selected peer-to-peer relay network has one or more member peer systems," recited in claim 24. 

14. The function call on page 27 of Exhibit C and reproduced as Exhibit C8 causes the 
code to provide bookkeeping of the network address of individual member peer systems to the 
underlying implementation of the peer relay system, which implements "providing network 
addresses of each of said one or more member peer systems to said joining peer system," recited 
in claim 24. 

15. The function call on page 22 of Exhibit C and reproduced as Exhibit C7 causes the 
code to enable a local client to join a relay grid, which implements "receiving a connection 
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update from said joining peer system indicating to which member peer systems said joining peer 

system is connected," recited in claim 24. 

1 6. The function call on page 22 of Exhibit C and reproduced as Exhibit C7 causes the 

code to start a sequence of actions and message exchanges, which implements therein each 
member peer system is connected to a number of other member peer systems that is less.man or 
equaj toa connection limit and each «^v^^^ m ^^„ m ^^^- 
for relaying data to the other member peer system, connected to that member peer system," 
recited in claim 24. r "\ 

17. Asevidencedb ya ^^ 
invention was reduced to practice prior to April 9, 2002. 

We each hereby declare that all statements made herein of our own knowledge ^fiue 
and that all statements made on information and belief are believed to be true; and taflgta 
these statements were made with the knowledge that willful false statements and the%e'fo 
made are punishable by fine or imprisonment, or both, under Section 1001 of Title rftf % ' 
United States Code and that such willful false statements may jeopardize the validity of the- 
application, or any patent issued thereon. 




Date ~+ 



-Anthony Mai 



Print or Typed name of Declarant 



«.ro i.'iiv! 

Signature of Declarant Date " 

Print or Typed name of Declarant 
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update from said joining peer system indicating to which member pear systems said joining peer 
system is connected," recited in claim 24. 

1 6. The function call on page 22 of Exhibit C end reproduced as Exhibit C7 causes the 
code to start a sequence of actions and message exchanges, which implements "wherein each 
member peer lystera is connected to a number of other member peer systems that is less than or.,, 
equal to a connection limit and each member peer system stores a set of one or more relay rules . 
for relaying data to the other member peer systems connected to that member pe*sr system," 
recited in claim 24. 

1 7. As evidenced by attached Exhibits A through C, every element of my claimed^'"" ' ' 
invention was reduced to practice prior to April 9. 2002. 

We each hereby declare that all statements made herein of our own knowledge are true 
and that all statements made on information and belief are believed to be true; and further that 
these statements were made with the knowledge that willful false statements and the like so 
made are punishable by fine or imprisonment, or both, under Section 1001 of Trde 18 of C the 
United States Code and that such wfllft 
application or any patent issued Thereon. 



Signature of Declarant 
Anthony Mai 



name of Declarant 




CTIen Van p^f, 



Print or typed name of Declarant 
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Anthony Mai/SDPD/SCEA 



Glen Van 
Datta/ SDPD/SCEA 



The document 



Here is the doc file attackment . 



Anthony Mai 

Sony Computer Entertainment America 

http://www.scea.com (See attached file: AudioProtocol.doc) 



Multi-Channel Multi-Party Audio Streaming Protocol 



Introduction 

Audio streaming in the online game scenery is different from conventional VoIP 
application in a number of ways. First, conventional VoIP system has only one data 
source. It may have one data target, like in the case of internet telephone, or it may have 
one server and multiple data targets, like in the case of internet radio or other broadcast 
steaming. 

In the online game scenery, there could be multiple data sources (each player in a 
game may speak), and there could also be multiple data targets (each player in a game 
may also listen.) And there may not even be a central server to receive and re-distribute 
all the audio data. 

Due to network bandwidth limitation, a multi-channel Multi-party audio 
streaming protocol must be designed to allow multiple players to talk with each other 
over the network. 

Assumption of the protocol: 

The following assumptions will be made: 

1 . There will always be one session master among all players in a game. When 
the session master quits the game, it will be detected when the session master 
is absent and a new session master will be selected from the remaining 
players. 

2. The session master maintains a list of all the available audio channels, or 
audio rooms. And it gives authorization of each player in each room to speak. 
Each individual player also keeps a copy of the audio room list. 

3. The network grid. Each will be connected to no more than 3 other 

■ directly. Any data initially received by a f rom on e of the 3 
it connects to will be forwarded to the two other IHH. Repeated data 
is not forwar ded. B y this mechanism, data from any one^m can eventually 
spread to all i n trie grid. 

The network grid 

The grid limits each network bandwidth requirement (sinc e it on ly needs 

to communicate with 3 other^^H) while allowing data from any single |H to 
quickly spread to every other ]^Hin the grid, using UDP sockets. 

There will be a network grid for the channel 0, or room #0. All are 
connected to this grid. This will be used for none-audio control messages. 

There will be one separate network grid for ea ch audio room. Each HI can 
optionally join a specific audio room. But each HI can join no more than one audio 
room at a time. 
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» connected to 3 otherMB, forming a grid ' 



Wi thin each audio room grid, there can be only one player speaking at a time. 
Any H wishing to speak should wait until the speaker has finished or paused. If 
multiple players try to speak, the collision will be detected and every one stops, and after 
a random time interval attempts to speak again. 

Establishment of the audio room network grid 

Any can create an audio room and this event is broadcast to every one in the 
game, through the channel 0 grid. The that created the audio room becomes the 
first node of the network grid. It will notice that it has three arms of connection, none 
occupied yet. 

Any wis hing to join an audio room broadcast a m essage through the 
channel 0 grid. Every within the grid then sends the new |H a welcome 

message, specifying whether they have a free arm of connection available or not. 

The new f" irst ta ^ QS offers of connection from existing^H| who has a free 
arm of connection. And then request connection with the very first |^Hwho greeted it 
with a n acknowledgement that it has no spare connection. Upo n such request, the existing 
HI breaks one of its connections and connects with the new H- 

Any with a brok en dow n connection will then announce the availability of 
connection on channel 0. The KB wno a ^ so ^ ave ^ Ge connection arm will respond to 
such announcements, and so the two establishes a connection. 



Establishment of the channel 0 grid 

Channel 0 grid is the net work grid that every |BI * s connected to. So it is 
important that when each H joins the game, it connects to the channel 0 grid properly. 

1. First H 

When the first creates a new g ame se ssion, there are no other BH- So 
all three grid connections of the first |B| is available. 





Second HH joins the game. Hie game server notifies 
firstflH about the second T he first sends 
a welcome message to second HI 311(1 tne y connect 



Third joins 




ThirdHH joins the game. The game server notifies 
first 2 about the thircflH. The first 2 ■■ 

send a welcome message to the third and they connect. 



Fourth | 




Fourth HH joins the game. The game server notifies 
existing ■■about the third Each existin g 
m sends a welcome message to the new^B and 

they connect. 



The new 



| joining protocol: 



1 .The game ser ver sen ds the new H info to all existing HH withi n a game. 

2. Each existing H sends a welcome UDP message to the new H. indicating 
whether it has any spare connection arm available. 

3. The new ^H respond to the first 3 (or less) welcome messages that indicated 
the availa bility of connection arms, by sending a connection request message to 
the MMdirecfly. 

4. TheUB that receives the connection request message sends a connection 
accepted messag e back. And the connection is established. 

5. If the new ^H still has 2 or more connection arms un used, it further sends a 
connection request message to the first existing |HI wn o sent a welcome 
message with no available connection a rm. 

6. Upon connection requests from the new an existing ^H would 
randomly break one of the connect ion ar ms, and then sends a connection 
accepted message back to the new HB 



Maintaining the grid 

| could drop out of the network grid unexpectedly, without notice. So it is 
important that the grid can recover from lost connection. 

1 . Each pair of connected HH sen d a Pi n S to eacn other periodically. 
R eceivin g a ping indicates that a connection is still al ive. 

2. If A is informed of disconnection from a nother |HL or if a ping has 
not been received for a certain time period, the HI marks that connection 
arm as free, and send out a connection arm available message, on channel 0. 

3. When ^H B receives a connection arm available message, and it has a free 
conne ction a rm, it responds by sending a connection request message. 

4. When HI A receives a connection request message, it sends a connection 
accepted message back. And the two is connected. 



Dropping out of the grid 

If a m intend to quit the game, it should post a message to all HI it 
connects to indicating that it is quitting. 

When a ping hasn't been received for certain time interval from a connected 
Hi; one h as t0 assume the connection no longer exist, and should then broadcast 
through th e exist ing arms of connection that it has a free arm of connection available. 
Any other HI wno a ^ so has a free arm of connection will then respond and so the two 
can connect. 



Transferring data within the grid 



The network grid is established to allow all |HI within the grid to share 
information with minimal network load for each individual ^^|. 

A sender ID and a sequence number uniquely identifies each package of the data, 
so when a m recei ves th e package, it knows whether it has already received the 
package or not. If the B| hasn 't received the package before, it will forward the 
package to the othertwo(B( it connects to, except for the one that it just received the 
package from. If a receives the same package again, it simply discard it, preventing 
the package from further circulation within the grid. 

This way, there will be no repeated packa ges received, and chances of lost 
package will be extremely low, since eachM| is connected to three other and 
the packaged could be forwarded to the BBfvia different paths. This way, the network 
grid concept addresses two of the common problems related to UDP sockets: duplicated 
or lost packages. 

Transferring audio data within the grid 

Each package of audio data will contain one or more frames of audio data. The 
package header will indicate the compression scheme, frame sequence number, frame 
count, sender of th e pack age, and whether it is a voiced or silence frame. 

When each f///^ within the grid receives the audio data package, forward the 
package according to the network grid protocol. At the same time it tries to sequence the 
audio data in prope r order, decode them, and send the data to audio output device. 

Each m decide whether it can speak or not according to the following rules: 

1 . If no voice package was received within certain time period (like 1 -2 
seconds), it can start to speak. 

2. If one or several silence voice packages are re ceived , that means the 
previous speak er has stopped speaking, so the m can start to speak. 

3. If after a starts to speak, it receives a voice package from a 
differ ent B, a collision has occurred. Under such circumstance the 

11 should immediately stop speaking, and wait for the next 
opportuni ty to sp eak. 

4. When the m can start to speak, there will be some indication on the 
computer screen hinting that the human player can start to speak. The 
player may or may not speak. If the player speaks, it will be 
automatically detected and the audio streaming engine will begin 
pumpin g audio data to the network grid. 

5. When a speaks, silence will be automatica lly dete cted and a 
silence p ackag e will be send out, allowing other to speak, while 
the localj^l will wait for next opportunity to speak. 

6. If a ^Hfspeaks for to o long (like 1 minutes), it will be forcefully 
stopped, allowing other Bfl m opportunity to start speaking. 
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:ription: Update the m_bSpeakAllowed status flag. This function is time 
dependent . 
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* Function: 
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Description: Return a sequence number for audio data. Note it is dif ferent 
from the sequence number returned by | 
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* Description: Send a chunk of data by UDP to a 



address . 





Description: Receive any incoming data on the UDP port. 
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Find a grid associated with a specific channel number. 
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{ 



|fo) ) 



} 






Page 25 




Page 26 




* Description: Create and add a new CPlayer object. 



* Returns : 




specific player. 
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